Getting started with NEON data: https://www.neonscience.org/resources/getting-started-neon-data-resources
Contact us form: https://www.neonscience.org/about/contact-us
Teaching Modules: https://www.neonscience.org/resources/learning-hub/teaching-modules
QUBES modules: https://qubeshub.org/community/groups/neon/educational_resources
EDDIE modules : https://serc.carleton.edu/eddie/macrosystems/index.html
Spatial data and maps: https://neon.maps.arcgis.com/home/index.html
NEON data portal: https://data.neonscience.org/
NEONScience GitHub repo: https://github.com/NEONScience
SFS 2025 NEON
Workshop GitHub repo: https://github.com/NEONScience/WORKSHOP-SFS-2025
This tutorial covers downloading NEON Aquatic Instrument System (AIS) and Aquatic Observation System (AOS) data using the neonUtilities R package, as well as basic instruction in beginning to explore and work with the downloaded data. This includes navigating data documentation, combining data within and between data products, visualizing data, and conducting qualitative and quantitative analysis with integrated AIS and AOS data.
The most popular function in neonUtilities is
loadByProduct(). This function downloads data from the NEON
API, merges the site-by-month files, and loads the resulting data tables
into the R environment, assigning each data type to the appropriate R
class. This is a popular choice because it ensures you’re always working
with the most up-to-date data, and it ends with ready-to-use tables in
R. However, if you use it in a workflow you run repeatedly, keep in mind
it will re-download the data every time.
Before we get the NEON data, we need to install (if not already done) and load the neonUtilities R package, as well as other packages we will use in the analysis.
# # Install neonUtilities package if you have not yet.
# install.packages("neonUtilities")
# install.packages("neonOS")
# install.packages("tidyverse")
# install.packages("plotly")
# Set global option to NOT convert all character variables to factors
options(stringsAsFactors=F)
# Load required packages
library(neonUtilities)
library(neonOS)
library(tidyverse)
library(plotly)
The inputs to loadByProduct() control which data to
download and how to manage the processing. The following are frequently
used inputs:
dpID: the data product ID, e.g. DP1.20288.001site: defaults to “all”, meaning all sites with
available data; can be a vector of 4-letter NEON site codes, e.g.
c("MART","ARIK","BARC").startdate and enddate: defaults to NA,
meaning all dates with available data; or a date in the form YYYY-MM,
e.g. 2017-06. Since NEON data are provided in month packages, finer
scale querying is not available. Both start and end date are
inclusive.package: either basic or expanded data package.
Expanded data packages generally include additional information about
data quality, such as individual quality flag test results. Not every
NEON data product has an expanded package; if the expanded package is
requested but there isn’t one, the basic package will be
downloaded.release: The data release to be downloaded; either
‘current’ or the name of a release, e.g. ‘RELEASE-2021’. ‘current’
returns provisional data in addition to the most recent release. To
download only provisional data, use release=‘PROVISIONAL’. Defaults to
‘current’. See https://www.neonscience.org/data-samples/data-management/data-revisions-releases
for more information.include.provisional: Should provisional data be
included in the downloaded files? Defaults to F.timeIndex: defaults to “all”, to download all data; or
the number of minutes in the averaging interval. See example below; only
applicable to IS data.check.size: T or F; should the function pause before
downloading data and warn you about the size of your download? Defaults
to T; if you are using this function within a script or batch process
you will want to set this to F.token: this allows you to input your NEON API token to
obtain faster downloads. Learn more about NEON API tokens in the
Using
an API Token when Accessing NEON Data with neonUtilities
tutorial.There are additional inputs you can learn about in the Use the neonUtilities R Package to Access NEON Data tutorial.
The dpID is the data product identifier of the data you
want to download. The DPID can be found on the
Explore Data Products page.
It will be in the form DP#.#####.###. For this tutorial, we’ll use some data products collected in NEON’s aquatics program:
Now it’s time to consider the NEON field site of interest. If not specified, the default will download a data product from all sites. The following are 4-letter site codes for NEON’s 34 aquatics sites as of 2025:
ARIK = Arikaree River CO
BARC = Barco Lake FL
BIGC = Upper Big Creek CA
BLDE = Black Deer Creek WY
BLUE = Blue River OK
BLWA = Black Warrior River AL CARI = Caribou Creek AK
COMO = Como Creek CO
CRAM = Crampton Lake WI
CUPE = Rio Cupeyes PR
FLNT = Flint River GA
GUIL = Rio Yahuecas PR HOPB = Lower Hop Brook MA
KING = Kings Creek KS
LECO = LeConte Creek TN
LEWI = Lewis Run VA
LIRO = Little Rock Lake WI
MART = Martha Creek WA MAYF = Mayfield Creek AL
MCDI = McDiffett Creek KS
MCRA = McRae Creek OR
OKSR = Oksrukuyik Creek AK
POSE = Posey Creek VA
PRIN = Pringle Creek TX
PRLA = Prairie Lake ND
PRPO = Prairie Pothole ND
REDB = Red Butte Creek UT
SUGG = Suggs Lake FL
SYCA = Sycamore Creek AZ
TECR = Teakettle Creek CA
TOMB = Lower Tombigbee River AL
TOOK = Toolik Lake AK
WALK = Walker Branch TN
WLOU = West St Louis Creek CO
In this exercise, we will pull data from NEON Atlantic Neotropical Domain (D04). The aquatic sites in D04 are Rio Cupeyes (CUPE) and Rio Yahuecas (GUIL). Just substitute the 4-letter site code for any other site at the end of the url.
Now let us download our data. We will focus our exercise on data
collected from 2021-10-01 through 2024-09-30 (water years 2022, 2023,
2024). If you are not using a NEON token to download your data,
neonUtilities will ignore the token input. We set
check.size = F so that the script runs well but remember
you always want to check your download size first. For this exercise, we
will focus on the following data products:
AIS Data Products:
AOS Data Products:
# download data of interest - AOS - Macroinvertebrate collection
inv <- neonUtilities::loadByProduct(dpID="DP1.20120.001",
site=c("CUPE","GUIL"),
startdate="2021-10",
enddate="2024-09",
package="basic",
release= "current",
include.provisional = T,
token = Sys.getenv("NEON_TOKEN"),
check.size = F)
The data we’ve downloaded comes as an object that is a named list of
objects. To work with each of them, select them from the list using the
$ operator.
# view all components of the list
names(inv)
## [1] "categoricalCodes_20120" "citation_20120_PROVISIONAL"
## [3] "citation_20120_RELEASE-2025" "inv_fieldData"
## [5] "inv_persample" "inv_taxonomyProcessed"
## [7] "issueLog_20120" "readme_20120"
## [9] "validation_20120" "variables_20120"
We can see that there are 10 objects in the downloaded macroinvertebrate collection data.
inv_fieldDatainv_persampleinv_taxonomyProcessedcategoricalCodes_20120issueLog_20120readme_20120validation_20120variables_20120citation_20120_PROVISIONALcitation_20120_RELEASE-2025If you’d like you can use the $ operator to assign an
object from an item in the list. If you prefer to extract each table
from the list and work with it as independent objects, which we will do,
you can use the list2env() function.
# unlist the variables and add to the global environment
list2env(inv,envir = .GlobalEnv)
## <environment: R_GlobalEnv>
Citing sources correctly helps the NEON user community maintain transparency, openness, and trust, while also providing a benefit of being able to track the impact of NEON on scientific research. Thus, each download of NEON data comes with proper citations custom to to the download that align with NEON’s data citation guidelines
# view formatted citations for DP1.20120.001 download
cat(citation_20120_PROVISIONAL)
## @misc{DP1.20120.001/provisional,
## doi = {},
## url = {https://data.neonscience.org/data-products/DP1.20120.001},
## author = {{National Ecological Observatory Network (NEON)}},
## language = {en},
## title = {Macroinvertebrate collection (DP1.20120.001)},
## publisher = {National Ecological Observatory Network (NEON)},
## year = {2025}
## }
cat(`citation_20120_RELEASE-2025`)
## @misc{https://doi.org/10.48443/rmeq-8897,
## doi = {10.48443/RMEQ-8897},
## url = {https://data.neonscience.org/data-products/DP1.20120.001/RELEASE-2025},
## author = {{National Ecological Observatory Network (NEON)}},
## keywords = {diversity, taxonomy, community composition, species composition, population, aquatic, benthic, macroinvertebrates, invertebrates, abundance, streams, lakes, rivers, wadeable streams, material samples, archived samples, biodiversity},
## language = {en},
## title = {Macroinvertebrate collection (DP1.20120.001)},
## publisher = {National Ecological Observatory Network (NEON)},
## year = {2025}
## }
# view the entire dataframe in your R environment
view(variables_20120)
There will always be one or more dataframes that include the primary data of the data product you downloaded. Multiple dataframes are available when there are related datatables for a single data product.
# view the entire dataframe in your R environment
view(inv_fieldData)
Using what you’ve learned above, can you modify the code to download data for the following data products?
# download data of interest - AIS - Continuous discharge
csd <- neonUtilities::loadByProduct(dpID="DP4.00130.001",
site=c("CUPE","GUIL"),
startdate="2021-10",
enddate="2024-09",
package="basic",
release= "current",
include.provisional = T,
token = Sys.getenv("NEON_TOKEN"),
check.size = F)
Let’s unpack the AIS data products too:
list2env(csd, .GlobalEnv)
## <environment: R_GlobalEnv>
Note that a few more objects were added to the Global Environment, including:
csd_continuousDischarge
The neonOS R package was developed to aid in working
with NEON Observational Subsystem (OS) data products. Two functions used
in this exercise are:
removeDups()joinTableNEON()Duplicates can arise in data, but the
neonOS::removeDups() function identifies duplicates in a
data table based on primary key information reported in the
variables_xxxxx files included in each data download.
Let’s check for duplicates in macroinvertebrate collection data
# what are the primary keys in inv_fieldData?
message("Primary keys in inv_fieldData are: ",
paste(variables_20120$fieldName[
variables_20120$table=="inv_fieldData"
&variables_20120$primaryKey=="Y"
],
collapse = ", ")
)
# identify duplicates in inv_fieldData
inv_fieldData_dups <- neonOS::removeDups(inv_fieldData,
variables_20120)
# what are the primary keys in inv_persample?
message("Primary keys in inv_persample are: ",
paste(variables_20120$fieldName[
variables_20120$table=="inv_persample"
&variables_20120$primaryKey=="Y"
],
collapse = ", ")
)
# identify duplicates in inv_persample
inv_persample_dups <- neonOS::removeDups(inv_persample,
variables_20120)
# what are the primary keys in inv_taxonomyProcessed?
message("Primary keys in inv_taxonomyProcessed are: ",
paste(variables_20120$fieldName[
variables_20120$table=="inv_taxonomyProcessed"
&variables_20120$primaryKey=="Y"
],
collapse = ", ")
)
# identify duplicates in inv_taxonomyProcessed
inv_taxonomyProcessed_dups <- neonOS::removeDups(inv_taxonomyProcessed,
variables_20120)
Thankfully, there are no duplicates in any of the AOS tables used in this exercise!
Every NEON data product comes with a Quick Start Guide (QSG). The QSGs contain basic information to help users familiarize themselves with the data products, including description of the data contents, data quality information, common calculations or transformations, and, where relevant, algorithm description and/or table joining instructions.
The QSG for Macroinvertebrate collection can be found on the data product landing page: https://data.neonscience.org/data-products/DP1.20120.001
The neonOS::joinTableNEON() function uses the table
joining information in the QSG to quickly join two related NEON data
tables from the same data product
# join inv_fieldData and inv_taxonomyProcessed
inv_fieldTaxJoined <- neonOS::joinTableNEON(inv_fieldData,inv_taxonomyProcessed)
Now, with field and taxonomy data joined. Individual taxon identifications are easily linked to field data such as collection latitude/longitude, habitat type, sampler type, and substratum class.
NEON often collects the same type of data from sensors in different
locations. These data are delivered together but you will frequently
want to plot the data separately or only include data from one sensor in
your analysis. NEON uses the horizontalPosition variable in
the data tables to describe which sensor data is collected from. The
horizontalPosition is always a three digit number for AIS
data. Non-shoreline HOR examples as of 2020 at AIS sites include:
The Continuous discharge data product is derived from a single
horizontalPosition, which corresponds to the sensor
co-located with the staff gauge at the site. This is also the location
at which all empirical discharge measurements are taken.
Let’s see from which horizontalPosition the Continuous
discharge data is published, and see if the other AIS data products we
downloaded have data at that same position.
# use dplyr from the tidyverse collection to get all unique horizontal positions
csd_hor <- csd_continuousDischarge%>%
dplyr::distinct(siteID,stationHorizontalID)
print(csd_hor)
## siteID stationHorizontalID
## 1 CUPE 110
## 2 GUIL 110
## 3 GUIL 132
# GUIL has two horizontal positions because the location of the staff gauge
# changed sometime during this time period. At what date did that occur?
max(csd_continuousDischarge$endDate[
csd_continuousDischarge$siteID=="GUIL"
&csd_continuousDischarge$stationHorizontalID=="110"
])
## [1] "2022-12-12 23:58:00 GMT"
At CUPE, the continuous discharge data are published from the 110 position, which is defined as ‘water level sensors mounted to a staff gauge at stream sites’.
At GUIL, until 2022-12-12, the continuous discharge data were published from the 110 position. On 2022-12-12, the position changed to 132, which is defined as ‘stand-alone water level sensors at downstream (S2) locations at stream sites.’
To make the continuous discharge data easier to work with for this
exercise, let’s use different packages from the tidyverse
collection to create a 30-min averaged table.
# 30-min average of continuous discharge data
CSD_30min <- csd_continuousDischarge%>%
dplyr::mutate(roundDate=lubridate::round_date(endDate,"30 min"))%>%
dplyr::group_by(siteID,roundDate)%>%
dplyr::summarise(dischargeMean=mean(continuousDischarge,na.rm=T),
dischargeCountQF=sum(dischargeFinalQFSciRvw,na.rm = T))
Notice that we included a summation of the quality flag (QF; binary: 1 = flag, 0 = no flag) fields in the new table.
Now that we have managed the data a bit to make it cleaner and easier to work with, let’s make some initial plots to see the AOS and AIS data separately before we begin to investigate questions that involve integrating the data.
First, we remove the records that were collected outside of normal sampling bouts as a grab sample, in cases where the NEON field ecologists saw interesting organisms that would not be captured using standard field sampling methods.
Next, we calculate macroinvertebrate abundance per square meter and taxon richness per sampling bout. This allows us to compare macroinvertebrate data among among different samplerTypes and habitatTypes.
# remove samples not associated with a bout
inv_clean <- inv_fieldTaxJoined%>%
dplyr::filter(is.na(samplingImpractical)
&!grepl("GRAB|BRYOZOAN",sampleID))
# calculate abundance - total count per m2
inv_clean$abun_M2 <- inv_clean$estimatedTotalCount/inv_clean$benthicArea
inv_clean <- inv_clean[order(inv_clean$abun_M2, inv_clean$scientificName),]
# summarize to get mean (+/- se) abundance by bout and sampler type
inv_abundance_summ <- inv_clean%>%
dplyr::rename(collectDate=collectDate.x)%>%
# dplyr::group_by(siteID,collectDate,eventID,sampleID,samplerType,boutNumber)%>%
dplyr::group_by(siteID,collectDate,eventID,sampleID,habitatType,boutNumber)%>%
dplyr::summarize(abun_M2_sum = sum(abun_M2, na.rm = TRUE))%>%
# dplyr::group_by(siteID,collectDate,eventID,samplerType,boutNumber)%>%
dplyr::group_by(siteID,collectDate,eventID,habitatType,boutNumber)%>%
dplyr::summarise_each(funs(mean,sd,se=sd(.)/sqrt(n())))%>%
dplyr::mutate(year=substr(eventID, 6,9),
yearBout=paste(year,"Bout",boutNumber, sep = "."))
# produce stacked plot to show trends within and across sites
inv_abundance_plot <- inv_abundance_summ%>%
# ggplot2::ggplot(aes(fill=samplerType, x=yearBout))+
ggplot2::ggplot(aes(fill=habitatType, x=yearBout))+
ggplot2::geom_bar(aes(y=abun_M2_sum_mean),
position=position_dodge(0.8), stat="identity")+
ggplot2::geom_errorbar(aes(ymin=abun_M2_sum_mean-abun_M2_sum_se,
ymax=abun_M2_sum_mean+abun_M2_sum_se),
width=0.4,colour="black",alpha=3.0,linewidth=0.2,
position=position_dodge(0.8))+
ggplot2::facet_wrap(~siteID,ncol = 1,scales="free_y")+
ggplot2::theme(axis.text.x = element_text(size = 10, angle = 30,
hjust = 1, vjust = 1))+
ggplot2::labs(title = "Mean macroinvertebrates per square meter",
y = "Abundance Per Square Meter",
x = "Bout")
inv_abundance_plot
# summarize to get mean (+/- se) richness by bout and sampler type
inv_richness_summ <- inv_clean%>%
dplyr::rename(collectDate=collectDate.x)%>%
# dplyr::group_by(siteID,collectDate,eventID,sampleID,samplerType,boutNumber)%>%
dplyr::group_by(siteID,collectDate,eventID,sampleID,habitatType,boutNumber)%>%
dplyr::count(siteID)%>%
dplyr::rename(count_per_samp = n)%>%
# group_by(siteID,collectDate,eventID,samplerType,boutNumber)%>%
group_by(siteID,collectDate,eventID,habitatType,boutNumber)%>%
dplyr::summarize(mean_count = mean(count_per_samp),
n=n(),
sd_count = sd(count_per_samp),
se=sd_count/sqrt(n))%>%
dplyr::mutate(year=substr(eventID, 6,9),
yearBout=paste(year,"Bout",boutNumber, sep = "."))
# produce plot to show trends within and across sites
inv_richness_plot <- inv_richness_summ%>%
# ggplot2::ggplot(aes(fill=samplerType, y=mean_count, x=yearBout))+
ggplot2::ggplot(aes(fill=habitatType, y=mean_count, x=yearBout))+
ggplot2::geom_bar(position="dodge", stat="identity")+
ggplot2::geom_errorbar(aes(ymin=mean_count-se, ymax=mean_count+se),
width=0.4, colour="black", alpha=3.0, linewidth=0.2,
position = position_dodge(0.8))+
ggplot2::facet_wrap(~siteID,ncol = 1)+
ggplot2::theme(axis.text.x = element_text(size = 10, angle = 30,
hjust = 1, vjust = 1))+
labs(title="Mean number of macroinvertebrate taxa per bout",
y= "Taxon Richness", x = "Bout")
inv_richness_plot
Let’s take a closer look at continuous discharge with the y-axis in log format. We will stack sites with one column to view the hydrographs for each site on the same timeseries.
CSD_plot <- CSD_30min%>%
ggplot2::ggplot(aes(x=roundDate,y=dischargeMean))+
ggplot2::geom_line()+
ggplot2::facet_wrap(~siteID,ncol = 1)+
ggplot2::scale_y_log10()+
labs(title="Continuous Discharge for Water Years 2022-2024",
y= "Discharge (L/s)", x = "Date")
CSD_plot
Next, we will use the R package plotly to make fun
interactive plots allowing us to view AOS and AIS data in the same
plotting field. There is a lot of code here to correctly format the plot
in a way to provide as much info and be as interactive as possible in a
single plotting field.
The plotly package allows us to interact with the plots
in the following ways:
# begin the plot code
AOS_AIS_plot_CUPE <- CSD_30min%>%
dplyr::filter(siteID=="CUPE")%>%
plotly::plot_ly()%>%
# add trace for continuous discharge
plotly::add_trace(x=~roundDate,y=~dischargeMean,
type="scatter",mode="line",
line=list(color = 'darkgray'),
name="Discharge")%>%
# add trace for INV abundance
plotly::add_trace(data=inv_abundance_summ%>%
dplyr::filter(siteID=="CUPE"),
x=~collectDate,y=~abun_M2_sum_mean,
# split=~paste0("INV Abundance: ",samplerType),
split=~paste0("INV Abundance: ",habitatType),
yaxis="y2",type="scatter",mode="markers",
error_y=~list(array=abun_M2_sum_se,
color='darkorange'),
marker=list(color="darkorange"),
visible="legendonly")%>%
# add trace for INV richness
plotly::add_trace(data=inv_richness_summ%>%
dplyr::filter(siteID=="CUPE"),
x=~collectDate,y=~mean_count,
# split=~paste0("INV Richness: ",samplerType),
split=~paste0("INV Richness: ",habitatType),
yaxis="y3",type="scatter",mode="markers",
error_y=~list(array=se,
color='darkgreen'),
marker=list(color="darkgreen"),
visible="legendonly")%>%
# define the layout of the plot
plotly::layout(
title = "CUPE Discharge w/ Macroinvertebrate Abundance & Richness",
# format x-axis
xaxis=list(title="dateTime",
automargin=TRUE,
domain=c(0,0.9)),
# format first y-axis
yaxis=list(
side='left',
title='Discharge (L/s)',
showgrid=FALSE,
zeroline=FALSE,
automargin=TRUE),
# format second y-axis
yaxis2=list(
side='right',
overlaying="y",
title='INV Abundance',
showgrid=FALSE,
automargin=TRUE,
zeroline=FALSE,
tickfont=list(color = 'darkorange'),
titlefont=list(color = 'darkorange')),
# format third y-axis
yaxis3=list(
side='right',
overlaying="y",
anchor="free",
title='INV Richness',
showgrid=FALSE,
zeroline=FALSE,
automargin=TRUE,
tickfont=list(color = 'darkgreen'),
titlefont=list(color = 'darkgreen'),
position=0.99),
# format legend
legend=list(xanchor = 'center',
yanchor = 'top',
orientation = 'h',
x=0.5,y=-0.2),
# add button to switch discharge between linear and log
updatemenus=list(
list(
type='buttons',
buttons=list(
list(label='linear',
method='relayout',
args=list(list(yaxis=list(type='linear')))),
list(label='log',
method='relayout',
args=list(list(yaxis=list(type='log'))))))))
AOS_AIS_plot_CUPE
# begin the plot code
AOS_AIS_plot_GUIL <- CSD_30min%>%
dplyr::filter(siteID=="GUIL")%>%
plotly::plot_ly()%>%
# add trace for continuous discharge
plotly::add_trace(x=~roundDate,y=~dischargeMean,
type="scatter",mode="line",
line=list(color = 'darkgray'),
name="Discharge")%>%
# add trace for INV abundance
plotly::add_trace(data=inv_abundance_summ%>%
dplyr::filter(siteID=="GUIL"),
x=~collectDate,y=~abun_M2_sum_mean,
# split=~paste0("INV Abundance: ",samplerType),
split=~paste0("INV Abundance: ",habitatType),
yaxis="y2",type="scatter",mode="markers",
error_y=~list(array=abun_M2_sum_se,
color='darkorange'),
marker=list(color="darkorange"),
visible="legendonly")%>%
# add trace for INV richness
plotly::add_trace(data=inv_richness_summ%>%
dplyr::filter(siteID=="GUIL"),
x=~collectDate,y=~mean_count,
# split=~paste0("INV Richness: ",samplerType),
split=~paste0("INV Richness: ",habitatType),
yaxis="y3",type="scatter",mode="markers",
error_y=~list(array=se,
color='darkgreen'),
marker=list(color="darkgreen"),
visible="legendonly")%>%
# define the layout of the plot
plotly::layout(
title = "GUIL Discharge w/ Macroinvertebrate Abundance & Richness",
# format x-axis
xaxis=list(title="dateTime",
automargin=TRUE,
domain=c(0,0.9)),
# format first y-axis
yaxis=list(
side='left',
title='Discharge (L/s)',
showgrid=FALSE,
zeroline=FALSE,
automargin=TRUE),
# format second y-axis
yaxis2=list(
side='right',
overlaying="y",
title='INV Abundance',
showgrid=FALSE,
automargin=TRUE,
zeroline=FALSE,
tickfont=list(color = 'darkorange'),
titlefont=list(color = 'darkorange')),
# format third y-axis
yaxis3=list(
side='right',
overlaying="y",
anchor="free",
title='INV Richness',
showgrid=FALSE,
zeroline=FALSE,
automargin=TRUE,
tickfont=list(color = 'darkgreen'),
titlefont=list(color = 'darkgreen'),
position=0.99),
# format legend
legend=list(xanchor = 'center',
yanchor = 'top',
orientation = 'h',
x=0.5,y=-0.2),
# add button to switch discharge between linear and log
updatemenus=list(
list(
type='buttons',
buttons=list(
list(label='linear',
method='relayout',
args=list(list(yaxis=list(type='linear')))),
list(label='log',
method='relayout',
args=list(list(yaxis=list(type='log'))))))))
AOS_AIS_plot_GUIL
Now, we can take what we have learned about NEON AOS and AIS data and look at other case studies using different NEON data products.